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Abstract 


A  survey  has  been  completed  to  investigate  literature  in  the  area  of  operator  performance 
using  multiple  displays,  in  order  to  support  a  variety  of  projects  in  the  Maritime  Command, 
Control,  Communications,  Computer  and  Intelligence  (C4I)  environment  at  Defence  Research 
and  Development  Canada  (DRDC)  -  Atlantic.  Using  a  defined  set  of  keywords  and  literature 
sources,  various  documents,  reports,  papers  and  articles  were  found,  of  which  100  were 
assessed  according  to  a  set  of  criteria  that  identified  those  that  were  more  useful  for  adaptation 
to  the  present  project’s  goals  and  objectives.  Of  the  literature  identified  in  the  survey,  7  were 
reviewed  in  detail  and  summarized. 


Resume 


Une  etude  des  documents  portant  sur  le  rendement  des  operateurs  qui  travaillent  avec  des 
affichages  multiples  a  ete  menee  dans  le  but  d’appuyer  une  variete  de  projets  de 
commandement,  de  controle,  de  communications,  d’informatique  et  de  renseignement  (C3IR) 
dans  le  milieu  maritime  a  Recherche  et  developpement  pour  la  defense  Canada  -  Atlantique 
(RDDC).  A  l’aide  d’un  ensemble  defini  de  mots  cles  et  de  sources  documentaires,  nous  avons 
trouve  divers  documents,  rapports,  joumaux  et  articles.  Nous  avons  evalue  100  de  ces 
documents  en  fonction  d’un  ensemble  de  criteres  qui  ont  permis  de  cerner  ceux  qui 
s’adapteraient  le  mieux  aux  objectifs  et  aux  huts  du  projet  en  cours.  A  partir  des  documents 
retenus,  nous  en  avons  examine  7  en  detail  et  les  avons  resumes. 
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Executive  summary 


Literature  survey  of  operator  performance  using  multiple 
displays 

Delle  Donne,  V.;  DRDC  Atlantic  CR  2005-209;  Defence  R&D  Canada  - 
Atlantic,  March  2006. 


Introduction  or  background 

This  document  is  the  final  report  of  a  project  to  investigate  literature  in  the  area  of  operator 
performance  using  multiple  displays,  in  order  to  support  a  variety  of  projects  in  the  Maritime 
Command,  Control,  Communications,  Computer  and  Intelligence  (C4I)  environment.  The 
report  has  been  prepared  by  Valcom  Consulting  Group  Inc.  for  the  Defence  Research  and 
Development  Canada  (DRDC)  -  Atlantic  under  Public  Works  and  Government  Services 
Canada  (PWGSC)  Contract  No.  W7707-05-2970. 


Results 

Using  a  defined  set  of  keywords  and  literature  sources,  various  documents,  reports,  papers  and 
articles  were  found,  of  which  1 00  were  assessed  according  to  a  set  of  criteria  that  identified 
those  that  were  more  useful  for  adaptation  to  the  present  project’s  goals  and  objectives.  Of 
the  literature  identified  in  the  survey,  7  were  reviewed  in  detail  and  summarized. 

The  type  of  literature  identified  indicated  that  there  is  more  relevant  literature  in  the  military 
than  the  maritime  domain.  There  is  also  a  substantial  base  of  literature  from  basic  research, 
which  is  directed  mostly  at  visual  displays.  This  literature  base,  however,  does  not  have  much 
depth.  Much  of  the  literature  is  at  the  experimental  phase  of  development.  This  reflects  the 
lack  of  maturity  of  the  domain,  in  that  fewer  new  ideas  have  been  generated  than  have  been 
tested. 

In  cases  where  actual  evaluations  of  multiple  display  technologies  have  been  conducted  there 
are  a  number  of  concerns  with  the  quality  of  research,  particularly  as  a  result  of 
unrepresentative  subject  groups  and  tasks,  inappropriate  measures,  and  poor  interface  design 
of  displays. 


Military  significance 

A  serious  challenge  for  operational  C4I  systems  is  how  operators  deal  with  extensive  amounts 
of  information  and,  accordingly,  the  complexity  of  operator-machine  interface  displays.  It  is 
anticipated  that  the  C4I  system  operator  of  the  future  will  perform  multiple  tasks  and  monitor 
and  attend  to  several  displays  concurrently.  Understanding  the  issues  related  to  human 
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performance  using  multiple  displays  is  key  to  providing  appropriate  and  effective  tools  and 
systems  to  best  support  tasks  of  the  user  in  the  military  environment. 


Future  plans 

To  address  weaknesses  in  the  literature  and  develop  guidance  for  future  operator  performance 
using  multiple  displays  in  the  maritime  environment,  it  is  recommended  that  a  research  and 
development  program  be  conducted,  as  follows: 

1 .  Extend  the  literature  review  on  operator  performance  using  multiple  displays; 

2.  Conduct  maritime  multiple  display  experiments;  and 

3.  Coordinate  and  consolidate  multiple  display  research. 


IV 
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Sommaire 


Etude  documentaire  sur  le  rendement  des  operateurs 
travaillant  avec  des  affichages  multiples 

Delle  Donne,  V.;  DRDC  Atlantic  CR  2005-209;  R  &  D  pour  la  defense  Canada 
-Atlantique;  March  2006. 


Introduction  ou  contexte 

Le  present  document  constitue  le  rapport  final  du  projet  visant  a  examiner  les  documents 
portant  sur  le  rendement  des  operateurs  qui  travaillent  avec  des  affichages  multiples  dans  le 
but  d’appuyer  une  variete  de  projets  de  commandement,  de  controle,  de  communications, 
d’informatique  et  de  renseignement  (C3IR)  dans  le  milieu  maritime.  Le  rapport  a  ete  prepare 
par  Valcom  Consulting  Group  Inc.  pour  Recherche  et  developpement  pour  la  defense  Canada 
-  Atlantique  (RDDC)  dans  le  cadre  du  contrat  n°  W7707-05-2970  avec  Travaux  publics  et 
Services  gouvemementaux  Canada  (TPSGC). 


Resultats 

A  l’aide  d’un  ensemble  defini  de  mots  cles  et  de  sources  documentaires,  nous  avons  trouve 
divers  documents,  rapports,  joumaux  et  articles.  Nous  avons  evalue  100  de  ces  documents  en 
fonction  d’un  ensemble  de  criteres  qui  ont  permis  de  cemer  ceux  qui  s’adapteraient  le  mieux 
aux  objectifs  et  aux  buts  du  projet  en  cours.  A  partir  des  documents  retenus,  nous  en  avons 
examine  7  en  detail  et  les  avons  resumes. 

De  plus,  le  type  de  documents  recenses  a  indique  qu’il  y  a  davantage  de  documents  pertinents 
dans  le  domaine  militaire  que  dans  le  domaine  maritime.  II  existe  aussi  un  nombre  important 
de  documents  crees  a  partir  de  recherches  fondamentales,  et  dont  le  sujet  porte  generalement 
sur  les  affichages  multiples;  toutefois,  ce  sujet  n’y  est  pas  vraiment  traite  en  profondeur.  La 
majorite  de  ces  documents  traitent  de  points  a  l’etape  experimental  de  leur  developpement, 
ce  qui  illustre  le  manque  de  maturite  du  domaine  (c.-a-d.  que  le  nombre  de  nouvelles  idees 
generees  est  inferieur  au  nombre  de  nouvelles  idees  testees). 

Dans  les  cas  ou  de  reelles  evaluations  de  la  technologie  des  affichages  multiples  ont  ete 
effectuees,  la  qualite  de  la  recherche  suscite  bon  nombre  d’inquietudes  en  raison,  plus 
particulierement,  des  groupes  et  des  taches  peu  representatifs,  des  mesures  inappropriees  ainsi 
que  de  la  conception  deficiente  de  f  interface  des  affichages. 
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Importance  militaire 


La  maniere  dont  les  operateurs  traitent  les  enormes  quantites  d’ informations  et,  par 
consequent,  la  complexite  des  affichages  de  l’interface  operateur-machine  represente  un  grave 
defi  pour  les  systemes  C4I.  II  est  prevu  que  l’operateur  des  systemes  C4I  de  l’avenir  assumera 
des  taches  multiples  et  surveillera  et  s’occupera  de  plusieurs  affichages  en  meme  temps.  II  est 
essentiel  de  comprendre  les  problemes  lies  a  la  performance  humaine  ou  il  est  question  de 
multiples  affichages  si  on  veut  offfir  des  outils  et  des  systemes  appropries  et  efficaces  pour 
mieux  soutenir  les  taches  de  l’utilisateur  dans  l’environnement  militaire. 


Perspectives 

Afm  de  s’attaquer  aux  lacunes  entourant  les  documents  et  de  developper  une  directive  pour  le 
rendement  des  futurs  operateurs  travaillant  avec  des  affichages  multiples  dans  le  milieu 
maritime,  nous  recommandons  qu’un  programme  de  recherche  et  de  developpement  soit  mis 
en  place  en  fonction  des  parametres  suivants  : 


1.  Elargir  l’examen  des  documents  sur  le  rendement  des  operateurs  qui  travaillent 
avec  des  affichages  multiples. 

2.  Effectuer  des  experiences  connexes  aux  affichages  multiples  dans  le  domaine 
maritime. 

3.  Coordonner  et  regrouper  les  recherches  sur  les  affichages  multiples. 


VI 
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1.  Introduction 


This  document  is  the  final  report  of  a  project  to  investigate  literature  in  the  area  of  operator 
performance  using  multiple  displays,  in  order  to  support  a  variety  of  projects  in  the  Maritime 
Command,  Control,  Communications,  Computer  and  Intelligence  (C4I)  environment.  The 
report  has  been  prepared  by  Valcom  Consulting  Group  Inc.  for  Defence  Research  and 
Development  Canada  (DRDC)  -  Atlantic  under  Public  Works  and  Government  Services 
Canada  (PWGSC)  Contract  No.  W7707-05-2970. 

1.1  Background 

A  serious  challenge  for  operational  C4I  systems  is  how  operators  deal  with  extensive  amounts 
of  information  available  to  them  and,  accordingly,  the  complexity  of  operator-machine 
interface  displays.  As  technology  produces  more  advanced  sensors,  and  the  ability  to  extract 
and  distribute  data  becomes  more  efficient  and  achievable,  the  quantity  of  information 
delivered  to  an  operator  will  continue  to  increase.  It  is  anticipated  that  the  C4I  systems 
operator  of  the  future  will  perform  multiple  tasks  and  monitor  and  attend  to  several  displays 
concurrently  using  systems  that  draw  on  visual,  auditory,  and  haptic  modalities. 

Spatially  distributing  information  across  several  displays  extends  the  area  to  which  the 
operator  must  attend.  Research  in  human  attention  suggests  that  performance  can  suffer  when 
attention  must  be  deployed  and  maintained  over  a  wide  field  of  view.  With  future  C4I 
systems  in  mind,  interface  design  must  provide  easily  accessible  decision  support  to  operators 
performing  multiple  tasks  and  monitoring  and  attending  to  multiple  information  sources. 

A  research  program  to  investigate  human  performance  using  multiple  displays  in  the  context 
of  Maritime  C4I  is  being  developed  at  DRDC  Atlantic.  The  program  will  explore  theoretical 
issues  surrounding  attentional  limitations  and  operator  workload  using  multiple  displays  in 
support  of  emerging  and  existing  Maritime  projects. 

To  support  a  variety  of  projects  in  the  Maritime  C4I  environment  DRDC  Atlantic  has 
requested  an  in-depth  overview  of  research  that  has  been  conducted  in  the  area  of  human 
performance  using  multiple  displays. 

1.2  Purpose 

The  overall  intentions  of  the  literature  survey  were  to: 

4.  Review  the  open  literature  and  collect  knowledge  from  other  practitioners  in  the 
area  of  human  performance  using  multiple  displays,  including  but  not  limited  to 
attentional  limitations,  sustained  attention,  and  workload;  and 

5.  Obtain  an  appreciation  of  performance  issues  and  areas  for  future  research  with 
respect  to  operator-machine  interfaces  in  the  Maritime  C4I  environment. 
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1.3  Scope 


The  scope  of  this  project  was  established  through  the  Statement  of  Work  developed  by  the 
Scientific  Authority,  combined  with  limits  placed  on  the  literature  that  could  be  reviewed 
within  the  project  budget. 

The  project  was  requested  to  be  completed  in  the  following  steps: 

A  thorough  search  of  the  literature  dealing  with  operator  performance  using  multiple  displays; 

A  critical  analysis  of  the  literature  results  and  methodologies,  assessing  its  strengths  and 
limitations  of  existing  research,  and  describing  the  data  collection,  analysis,  and  reporting 
techniques; 

A  determination  of  the  current  knowledge  and  problems  of  human  performance  when  using 
multiple  displays; 

Identification  of  current  strategies  used  by  operators  in  various  industries,  as  well  as  military; 
and 

The  formulation  of  conclusions  and  recommendations  consistent  with  the  goals  of  the  survey. 

In  order  to  remain  within  the  project  budget  it  was  determined  that  up  to  approximately  150 
documents,  reports,  papers  and/or  articles  would  be  entered  into  the  project  database,  while 
approximately  7  would  be  reviewed  in  detail. 

1.4  Deliverables 

The  deliverables  required  from  this  literature  survey  project  included: 

Annotated  biography  summarizing  the  breath  and  depth  of  the  literature;  and 

Draft  and  final  reports  to  further  summarize  and  assess  the  relevance  of  the  literature  findings, 
and  determine  the  future  research  and  development  activities  that  may  be  required. 
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2.  Methodology 


The  literature  survey  was  conducted  through  the  methodology  outlined  in  the  following 
subsections. 

2.1  Literature  Survey  and  Preliminary  Review 

The  project  Statement  of  Work  called  for  a  survey  to  assess  the  breadth  and  depth  of  the 
immediately  available  literature  related  to  human  performance  using  multiple  displays.  This 
was  to  be  followed  with  an  assessment  of  the  relevance  of  the  literature  to  the  use  and  design 
of  Maritime  C4I  systems  and  provide  recommendations  for  future  research. 

Activities  included  literature  search  and  preliminary  review  in  order  to  classify  the  type  of 
literature  and  the  level  of  research. 

2.1.1  Strategy 

The  project  began  with  the  development  of  the  literature  search  strategy,  by  defining 
multiple  displays  for  the  purpose  of  the  project,  defining  operator  performance  impacts 
of  multiple  displays,  identifying  the  key  search  terms  required,  and  identifying  the 
sources  of  literature  that  should  be  searched. 

An  iterative  search  strategy  was  used,  such  that  literature  sources  would  be  searched  in 
multiple  passes,  identify  the  most  relevant  literature  on  the  first  pass,  and  then  return  to 
obtain  another  level  of  literature.  These  iterations  allowed  the  incremental 
identification  of  literature,  prioritized  by  project  objectives  while  at  the  same  time 
remaining  within  the  project  scope. 

2.1.2  Definitions 

An  operator-machine  system  can  be  considered  as  a  combination  of  one  or  more 
operators  and  one  or  more  physical  components  interacting  to  bring  about  from  given 
inputs  some  desired  out.  The  machine  can  consist  of  virtually  any  type  of  physical 
object,  device,  equipment,  facility,  thing,  or  what  have  you  that  operators  use  in 
carrying  out  some  activity  that  is  directed  toward  achieving  some  desired  purpose  or  in 
performing  some  function.  The  essential  nature  of  an  operator’s  involvement  in  a 
system  is  an  active  one,  interacting  with  the  system  to  fulfill  the  function  for  which  the 
system  is  designed.  The  typical  type  of  interaction  between  an  operator  and  a  machine 
is  illustrated  in  Figure  1  ( Schematic  Representation  of  an  Operator-machine  System ). 
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Figure  1.  Schematic  Representation  of  an  Operator-machine  System 


This  illustration  shows  how  the  displays  of  a  machine  serve  as  stimuli  for  an  operator, 
trigger  some  type  of  information  processing  on  the  part  of  the  operator  (  including 
decision  making),  which  in  turn  results  in  some  action  that  controls  the  operation  of  the 
machine. 

In  the  present  context,  the  term  display  refers  to  the  presentation  of  information  to  any 
of  the  human  senses,  not  necessarily  just  to  the  eyes.  The  purpose  of  the  display  in  a 
system  is  to  give  information  to  the  operator  about  the  functional  condition  of  the 
system  or  the  process.  The  information  can  be  categorized  as  follows:  need  to  know, 
nice  to  know  and  historical. 

Information  is  sometimes  displayed  in  a  confusing  way,  with  less  critical  data 
obscuring  the  presence  of  information  on  which  action  must  be  taken.  Because  the 
purpose  of  a  display  is  to  notify  the  operator  of  a  situation,  two  prime  concerns  are  to 
make  the  signal  easily  detectable  and  to  have  it  indicate  clearly  any  required  actions. 
There  are  several  modes  by  which  information  from  displays  is  conveyed  to  an 
operator:  visual,  auditory,  and  haptic  (tactile). 

Visual  presentation  is  preferred  for  complex  messages  in  noisy  environments  where 
response  time  is  not  critical.  Auditory  presentation  is  preferred  for  simple  messages  in 
areas  where  people  move  around  frequently  and  where  response  time  must  be  rapid. 
Usually  auditory  displays  supplement  visual  presentation  by  drawing  the  operator’s 
attention  to  the  visual  display  that  provides  detail  of  the  system.  Auditory  displays  are 
particularly  well  suited  to  represent  infrequently  occurring  information  where  it  is 
necessary  to  gain  the  operator’s  attention.  Haptic  (tactile)  modes  are  being  employed 
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more  often  as  people  are  bombarded  with  more  information,  especially  when  there  is 
high  ambient  noise  or  vision  is  obscured. 

Displays  of  a  system  can  either  facilitate  interaction  or  increase  task  difficulty  and  the 
probability  of  error.  The  German  Gestalt  psychologists  in  the  first  half  of  the  twentieth 
century  described  how  the  way  stimuli  are  structured  determines  how  they  are 
perceived.  The  structure  of  a  display  is  of  superior  importance  to  any  particular 
element  within  it.  Hence  the  Gestalt  dictum  “the  whole  is  greater  than  the  sum  of  its 
parts”.  Conscious  perceptions  result  not  just  from  an  analysis  of  objects  in  the  field  of 
view  (FOV)  but  from  a  synthesis  of  the  objects  themselves  and  the  relation  between 
them. 

Finally,  human  performance  measures  are  usually  frequency  measures  (such  as  number 
of  targets  detected),  latency  measures  (such  as  reaction  time  or  delay  in  switching  from 
one  activity  to  another),  duration  measures  (such  as  time  on  target  in  a  tracking  task)  or 
reliability  measures  (such  as  probability  of  errorless  performance).  Sometimes 
combinations  of  these  basic  types  are  used,  such  as  number  of  missed  targets  per  unit 
time. 

2.1.3  Keywords 

As  a  result  of  the  project  planning  activities  and  consultation  with  the  scientific 
community,  the  following  list  of  primary  keywords  were  used  in  combination  (i.e., 
multiple  AND  display)  in  the  literature  search: 

Performance; 

Operator  performance; 

Human  performance; 

Display;  and 

Multiple  displays. 

These  keywords  focused  on  the  concept  of  multiple  displays  and  some  of  the  key 
human  behaviour  (such  as  human  limitations  and  workload)  that  would  underlie 
performance  impacts  of  multiple  display  use. 

Other  keywords  were  used  for  basic  research  issues  that  likely  relate  to  multiple 
display  requirements,  such  as: 

Visual  display; 

Auditory  display; 

Haptic  (tactile)  display; 
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Human-machine  interface; 

Human-computer  Interaction; 

Operator-machine  interface; 

Visual  Search; 

Perception; 

Attention; 

Vigilance; 

Workload; 

Change  blindness; 

Attentional  blink; 

Attentional  limitations; 

Sustained  Attention; 

Attentional  spotlight;  and 
Field  of  view  (FOV). 

2.1.4  Databases 

The  literature  search  was  conducted  using  various  primary  sources. 

Firstly,  literature  was  obtained  by  searching  scientific  literature  databases  (including 
PsycINFO,  PsycARTICLES,  and  Compendex  to  name  a  few)  accessed  through  the 
Champlain  library  at  Universite  de  Moncton  in  New  Brunswick. 

Then  a  considerable  amount  of  literature  search  was  conducted  on-line,  using  Copernic 
desktop  search  engine.  Within  the  scientific  literature,  a  number  of  websites  and 
databases  were  searched  including: 

Association  of  Canadian  Ergonomists  (ACE)  (http://www.ace-ergocanada.ca) 

Association  of  Computing  Machinery  (ACM)  (http://www.acm.org) 

Association  of  Computing  Machinery  (ACM)  Portal  Digital  Library 
(http://portal.acm.org) 

Atomic  Energy  of  Canada  Limited  (AECL)  (http://www.aecl.ca) 
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Australian  Department  of  Defence,  Defence  Science  and  Technology  Organization 
(DSTO)  (http://www.dsto.defence.gov.au) 

Cognitive  Task  Analysis  (CTA)  Resource  (http  ://www.ctaresource.  com) 

Department  of  National  Defence  (DND),  Defence  Research  and  Development  Canada 
(DRDC)  (http://pubs.drdc-rddc.dnd.ca) 

Edinburgh  Engineering  Virtual  Library  (EEVL)  (http://www.eevl.ac.uk) 

Electronic  Power  Research  Institute  (EPRI)  (http://www.epri.com) 

Engineering  Information  (http://www.ei.org) 

Human-Computer  Interaction  (HCI)  Bibliography  (http://www.hcibib.org) 

Human  Factors  and  Ergonomics  Society  (HFES)  (http://www.hfes.org) 
IngentaConnect  (http://www.ingentaconnect.com) 

Lawrence  Erlbaum  Associates  (LEA)  Online  (http://www.leaonline.com) 

MetaPress  (http://www.metapress.com) 

National  Aeronautics  and  Space  Administration  (NASA),  Johnson  Space  Center,  Man- 
Systems  Integration  Standards  (MSIS)  (http://msis.isc.nasa.gov) 

National  Research  Council  Canada  (NRC),  Canada  Institute  for  Scientific  and 
Technical  Information  (CISTI)  (http://cisti-icist.nrc-cnrc.gc.ca) 

Pacific  Science  &  Engineering  Group  (http://www.pacific-science.com) 

QinetiQ  (http://www.qinetiq.com) 

U.S.  Nuclear  Regulatory  Commission  (NRC)  (http://www.nrc.gov) 

U.S.  Department  of  Defense  (DoD),  Defense  Technical  Information  Center  (DTIC), 
Directory  of  Design  Support  Methods  (DDSM)  -  MATRIS 
(h  ttp :  // w  w  w  .dtic.mil  /  m  at  r  i  s/d  d  s  m/) 

U.S.  Department  of  Defense  (DoD),  Defense  Technical  Information  Center  (DTIC), 
Human  Factors  Engineering  Technical  Advisory  Group  -  Standardization 
(http://hfetag.dtic.mil) 

U.S.  Department  of  Defense  (DoD),  Defense  Technical  Information  Center  (DTIC), 
Information  Analysis  Centers  (IAC),  Human  Systems  Information  Analysis  Center 
(HSIAC)  (http://iac.dtic.mil/hsiac/) 
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U.S.  Department  of  Defense  (DoD),  Space  and  Naval  Warfare  Systems  Command 
(SPAWAR)  (http://enterprise.spawar.navy.mil~) 

U.S.  Department  of  Transportation  (DoT),  Research  and  Special  Programs 
Administration  (RSPA),  John.  A.  Volpe  National  Transportation  Systems  Center, 
Operator  Performance  and  Safety  Division  (OPSAD) 
(http://www.volpe.dot.gov/opsad/) 


2.2  Literature  Categorization  and  Ranking  System 

Once  relevant  titles  were  obtained  they  were  further  reviewed  and  categorized  using  a 
matrix  illustrated  in  Table  1  ( Literature  Categorization  Matrix)  below. 


Table  1.  Literature  Categorization  Matrix 


LITERATURE  TYPE 

CONCEPT 

LAB 

FIELD 

REVIEW 

DOMAIN 

RELEVANCE 

MARITIME 

MILITARY 

INDUSTRY 

BASIC 

The  four  categories  of  Domain  Relevance,  from  lowest  to  highest  on  the  vertical  axis, 
included: 

1 .  Basic  Research; 

2.  Industry  Applications; 

3.  Military  Applications;  and 

4.  Maritime  Applications. 

The  four  categories  of  Literature  Type,  from  lowest  to  highest  on  the  horizontal  axis, 
included: 

1 .  Conceptual; 

2.  Laboratory  Experiment; 
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3.  Field  Evaluation;  and 

4.  Literature  Review. 

Once  relevant  titles  were  categorized  they  were  further  reviewed  and  assessed  using 
criteria  given  in  Table  2  ( Criteria  for  the  Critical  Analysis  -  Initial  Assessment)  below. 


Table  2.  Criteria  for  the  Critical  Analysis  -  Initial  Assessment 

REFERENCE: _  CATEGORY: 


CRITICAL  ANALYSIS  CRITERIA 

SCALE  STRUCTURE 

1 .  Topic’s  relationship  to  the  main  topic  of  interest  to  the  project  (see  Section  1 .2 
above). 

1 .  Indirectly  related 

2.  Directly  related 

3.  Exact  match 

2.  Level  of  detail  provided  regarding  the  methodology  and  the  data  collection 
techniques. 

1 .  Less  than  adequate 

2.  Adequate 

3.  More  than  adequate 

3.  Adequacy  of  research  design  to  answer  research  questions. 

1 .  Less  than  adequate 

2.  Adequate 

3.  More  than  adequate 

4.  Adequacy  of  the  chosen  methodology  to  obtain  reliable  data. 

1 .  Less  than  adequate 

2.  Adequate 

3.  More  than  adequate 

5.  Relevance  of  the  methodology  to  Maritime  C4I  environments  (validity). 

1 .  Indirectly  related 

2.  Directly  related 

3.  Exact  match 

6.  Reliability  of  the  results  based  on  sample  size,  data  collection  techniques, 
objectivity  of  data,  etc. 

1 .  Low 

2.  Medium 

3.  High 

7.  Level  of  contribution  to  the  body  of  knowledge  on  operator  performance  using 
multiple  displays. 

1 .  Low 

2.  Medium 

3.  High 

This  table  was  created  to  meet  one  of  the  project  goals,  which  was  to  provide  an 
indication  of  the  quality  of  the  literature.  Each  literature  was  assessed  for  each 
criterion,  and  the  totals  for  each  literature  were  tallied  and  entered  into  a  summary 
table,  and  assigned  a  ranking  as  shown  in  Table  3  (. Literature  Ranking )  below. 
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Table  3.  Literature  Ranking 


CRITICAL  ANALYSIS  CRITERIA 

LITERATURE 

1 

2 

3 

4 

5 

6 

7 

TOTAL 

Literature  1 

Literature  2 

Literature  3 

The  literature  was  then  arranged  by  rank  within  each  category,  and  the  highest-ranking 
literature  was  profiled  and  considered  for  a  detailed  review. 


2.3  Detailed  Literature  Review 

Once  the  literature  was  organized  into  its  respective  category  of  type,  7  of  the  1 00 
documents,  reports,  papers  and/or  articles  were  selected  for  a  more  detailed  review. 
This  selection  was  based  on  a  review  of  the  abstracts  of  the  literature  listed  in  the 

Bibliography. 

Once  the  detailed  literature  review  was  completed,  an  assessment  of  the  breadth,  depth 
and  quality  of  the  literature  was  developed,  along  with  recommendations  for  areas  of 
further  research  in  the  future.  These  final  assessments  were  integrated  into  this  final 
report. 
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3.  Summary  of  Findings  and  Discussion 


Using  the  keywords  and  literature  sources  outlined  in  Section  2.1  a  total  of  152  documents, 
reports,  papers  and/or  articles,  of  which  100  of  the  most  relevant  literature  have  been  chosen 
and  categorized  using  a  category  matrix,  to  produce  the  literature  distribution  illustrated  in 
Table  4  {Literature  Categorization  Results)  below. 

Table  4.  Literature  Categorization  Results 


LITERATURE  TYPE 

CONCEPT 

LAB 

FIELD 

REVIEW 

DOMAIN  RELEVANCE 

MARITIME 

2 

8 

1 

2 

MILITARY 

4 

8 

2 

12 

INDUSTRY 

2 

11 

1 

4 

BASIC 

9 

24 

1 

9 

Note  that  the  numbers  in  each  cell  indicate  the  number  of  documents  in  the  project  assigned  to 
that  domain  relevance  and  literature  type. 
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3.1  Literature  Categorization  Results 


The  relevant  literature  was  categorized  in  terms  of  domain  as  shown  in  Figure  2  {Literature 
by  Domain )  below. 


Maritime 

13% 


Industry 

18% 


Figure  2.  Literature  by  Domain 


The  above  figure  clearly  demonstrates  that  there  is  a  substantial  base  of  literature  from  Basic 
(43%)  research,  which  is  directed  mostly  at  visual  displays.  There  is  also  a  substantial  base  of 
literature  from  Military  (26%)  applications,  which  is  directed  mostly  at  aviation  cockpit 
displays.  Not  surprisingly,  the  number  of  Military  (26%)  work  is  double  the  literature  related 
directly  to  Maritime  (13%)  work.  But  there  is  surprisingly  less  literature  from  Industrial 
(18%)  domains,  such  as  transportation. 

The  relevant  literature  was  also  categorized  in  terms  of  type  as  shown  in  Figure  3  {Literature 
by  Type)  below. 
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Concept 

Review  1 7% 


51% 


Figure  3.  Literature  by  Type 


As  illustrated  above,  more  than  half  of  the  literature  identified  is  in  the  category  of  Laboratory 
(51%)  experiment.  This  reflects  the  lack  of  maturity  of  the  domain,  in  that  Concepts  (17%) 
have  been  generated  and  are  still  being  thoroughly  tested.  Even  further,  there  is  a  noticeable 
lack  of  Field  (5%)  evaluations  supporting  this  conclusion. 

Another  reason  for  the  heavy  weighting  of  literature  in  the  area  of  tested  concepts  is  that  the 
area  of  operator  performance  using  multiple  displays  is  very  much  a  technology-enabled  field. 
As  new  technologies  are  developed  in  an  extremely  fast  pace,  possible  forms  of  display 
support  also  change.  The  available  technologies  for  operator  performance  support  have  been 
developing  very  rapidly,  and  with  such  technological  expansion,  there  are  many  rewards  in 
the  publication  and  commercialization  for  testing  exciting  concepts.  This  pattern  is  seen  to  a 
large  extent  in  the  human-computer  interaction  literature  and  the  basic  research  category. 
However,  that  pattern  continues  in  the  military  and  industrial  categories,  with  a  heavy 
concentration  on  evaluation  in  the  laboratory  and  a  slower  movement  towards  new  concepts 
or  field  evaluations. 

3.2  Literature  Ranking  Results 

Of  the  literature  identified  in  the  survey,  and  summarized  in  Section  3.1,  a  number  of 
documents  were  selected  for  a  more  detailed  review.  This  list  was  developed  based  on  a 
review  of  the  abstracts  of  the  literature  listed  in  the  Bibliography.  Unfortunately,  the  limited 
budget  for  the  project  provided  time  for  a  review  of  only  7  best-ranked  documents  shown  in 
Table  4  (Literature  Ranking  Results)  below. 
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Table  4.  Literature  Ranking  Results 


LITERATURE 

TOTAL 

DiVita,  J.C.,  Obermayer,  R.,  Nugent,  W.,  and  Linville,  J.M.  (2004) 

17 

Durlach,  P.J.  (2004) 

17 

Hale,  K.S.  and  Stanney,  K.M.  (2003) 

17 

Samman,  S.N.,  Stanney,  K.M.  and  Sims,  V.K. 

16 

Stanney,  K.M.,  Samman,  S.N.,  Reeves,  L.,  Hale,  K.S.,  Buff,  W....(2004) 

16 

St.  John,  M.  Manes,  D.I.,  Oonk,  H.M.,  and  Ko,  H.  (1999) 

16 

Wickens,  C.D.,  Vincow,  M.A.,  Schopper,  A.W.,  and  Lincoln,  J.E.  (1997) 

16 

The  next  section  presents  a  short  summary  of  the  literature  and  brief  comments  on  how 
multiple  displays  apply  to  Maritime  C4I. 

3.3  Literature  Reviewed  in  Detail 

DiVita,  J.C.,  Obermayer,  R.,  Nugent,  W.,  and  Linville,  J.M.  (2004).  Verification  of  the 
Change  Blindness  Phenomenon  While  Managing  Critical  Events  on  a  Combat  Information 
Display.  Change  blindness  occurs  when  humans  are  unable  to  detect  significant  changes  in 
objects  and  scenes  after  their  attention  is  momentarily  diverted.  Because  change  blindness  is 
relevant  in  many  applied  settings,  the  study  investigated  the  phenomenon  in  the  context  of 
tasks  performed  by  naval  command  and  control  system  personnel.  Operators  of  such  systems 
are  often  heavily  loaded  with  concurrent  visual  search,  situation  assessment,  voice 
communications,  and  control  display  manipulation  tasks  at  large,  physically  dispersed  tactical 
situation  displays.  As  the  operators’  attention  shifts  from  one  display  to  another,  it  creates  an 
opportunity  for  changes  to  occur  on  unattended  screens  with  potentially  negative 
consequences.  The  results  show  that  on  a  display  containing  8  objects  of  interest, 
considerable  change  blindness  was  demonstrated  in  that  participants  required  2  or  more 
selections  to  correctly  identify  a  changed  object  on  nearly  1/3  of  the  test  trials.  Further, 
operator  performance  on  15%  of  the  trials  was  equivalent  to  randomly  guessing  with 
replacement  after  making  3  incorrect  selections.  This  research  underscores  the  need  for 
developing  effective  countermeasures  to  the  change  blindness  phenomenon.  Actual  or 
potential  uses  of  this  research  include  interface  design  of  computer  workstations  for  military 
applications. 

Durlach,  P.J.  (2004).  Change  Blindness  and  Its  Implications  for  Complex  Monitoring  and 
Control  Systems  Design  and  Operator  Training.  Recent  research  on  change  detection 
suggests  that  people  often  fail  to  notice  changes  in  visual  displays  when  they  occur  at  the 
same  time  as  various  forms  of  visual  transients,  including  eye  blinks,  screen  flashes,  and 
scene  relocation.  Distractions  that  draw  the  observer’s  attention  away  from  the  location  of  the 
change  especially  lead  to  detection  failure.  As  process  monitoring  and  control  systems  rely 
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on  humans  interacting  with  complex  visual  displays,  there  is  a  possibility  that  important 
changes  in  visually  presented  information  will  be  missed  if  the  changes  occur  coincident  with 
a  visual  transient  or  distraction.  The  purpose  of  this  article  was  to  review  research  on  so 
called  “change  blindness”  and  discuss  its  implications  for  the  design  of  visual  interfaces  for 
complex  monitoring  and  control  systems.  The  major  implication  is  that  systems  should 
provide  users  with  dedicated  change-detection  tools,  instead  of  leaving  change  detection  to 
the  vagaries  of  human  memorial  and  attentional  processes.  Possible  training  solutions  for 
reducing  vulnerability  to  change-detection  failure  are  also  discussed. 

Hale,  K.S.  and  Stanney,  K.M.  (2003).  Deriving  Haptic  Design  Guidelines  from  Human 
Physiological  Psychophysical  and  Neurological  Foundations.  In  order  to  realize  successful 
integration  of  haptic  interactions  into  multimodal  systems,  the  sensory,  perceptual  and 
cognitive  abilities  and  limitations  of  users  must  be  examined,  and  accounted  for  in  the  design 
of  haptic  systems.  To  date,  many  interactive  systems  have  focused  on  visual  cues,  and  to  a 
lesser  extent,  audio  cues,  for  presenting  information  to  users  concerning  their  surroundings 
and  interactions  with  objects.  The  haptic  sense  offers  another  independent  sensory  channel 
that  can  be  processed  in  the  brain  to  further  enhance  the  user’s  experience  in  a  multimodal 
environment.  By  adding  in  haptic  feedback,  realism  of  the  system  should  be  enhanced 
through  more  natural  interaction  with  objects  and  environmental  surroundings.  This  survey 
paper  begins  with  a  review  of  physiological  and  psychophysical  aspects  of  human  cutaneous 
and  kinesthetic  senses.  From  there,  issues  of  incorporating  haptics  interaction  into  a  visual 
display  are  discussed.  Based  on  an  exhaustive  literature  review,  the  paper  offers  many  haptics 
design  guidelines  to  aid  developers  of  multimodal  interactive  systems. 

Samman,  S.N.,  Stanney,  K.M.  and  Sims,  V.K.  Think  Multimodal  to  Maximize  Information 
management  Capacity.  New  human-computer  interaction  paradigms  are  being  developed  that 
transform  user  interaction  from  a  primarily  visual  experience  to  one  that  uses  multiple  senses. 
While  research  has  advanced  in  the  development  of  technological  breakthroughs,  relatively 
little  research  has  examined  the  cognitive  factors  of  the  user  during  multimodal  multitasking 
operations.  Thus,  designers  may  face  problems  when  creating  interfaces  that  allow  operators 
to  process  the  optimal  amount  of  data.  It  is  herein  proposed  that  more  effective  multimodal 
interaction  can  be  accomplished  by  creating  multimodal  display  systems  that  are  expressly 
designed  to  maximize  user  information  processing  across  the  sensory  systems.  Rather  than 
information  overload,  the  benefits  of  such  systematically  designed  multimodal  interaction 
should  be  greater  information  assimilation,  retention,  and  recall.  A  theoretically  derived 
multimodal  interaction  framework  is  proposed  that  leverages  multiple  sensory  systems  to 
maximize  working  memory  (WM)  throughput.  Multiple  resource  theory  (MRT)  suggests  that 
enhancements  in  human  information  management  capacity  may  be  realized  via  multimodal 
interaction.  The  current  study  proposes  an  expansion  of  the  traditional  bi-modal  (  verbal, 
visual/spatial)  model  of  WM  to  a  multimodal  WM  system,  which  includes  verbal,  visual, 
spatial,  kinesthetic,  tactile,  and  tonal  component  subsystems.  Single  modality  capacity  was 
measured  for  each  proposed  subsystem.  In  addition,  multimodal  capacity  was  calculated  for 
combined  modalities.  Results  demonstrated  that  multimodal  WM  capacity  surpasses  that  of 
single  modality  capacity.  Most  notably,  multimodal  WM  capacity  averaged  more  than  three 
times  the  traditional  capacity  limitation  of  unidimensional  WM.  Applications  for  the 
multimodal  WM  framework  are  relevant  when  designing  multimodal  systems  for  multitasking 
environments  such  as  those  conduced  by  air  traffic  controllers,  emergency  room  physicians, 
and  stock  brokers. 
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Stanney,  K.M.,  Samman,  S.N.,  Reeves,  L.,  Hale,  K.S.,  Buff,  W.,  Bowers,  C.,  Goldiez,  B, 
Nicholson,  D.,  and  Lackey,  S.  (2004).  A  Paradigm  Shift  in  Interactive  Computing:  Deriving 
Multimodal  Design  Principles  from  Behavioral  and  Neurological  Foundations.  As 
technology  advances,  systems  are  increasingly  able  to  provide  more  information  than  a  human 
operator  can  process  accurately.  Thus,  a  challenge  for  designers  is  to  create  interfaces  that 
allow  operators  to  process  the  optimal  amount  of  data.  It  is  herein  proposed  that  this  may  be 
accomplished  by  creating  multimodal  display  systems  that  augment  or  switch  modalities  to 
maximize  user  information  processing.  Such  a  system  would  ultimately  be  informed  by  a 
user’s  neurophysiological  state.  As  a  first  step  toward  that  goal,  relevant  literature  is  reviewed 
and  a  set  of  preliminary  design  guidelines  for  multimodal  information  systems  is  suggested. 

St.  John,  M.  Manes,  D.I.,  Oonk,  H.M.,  and  Ko,  H.  (1999).  Workspace  Control  Diagrams  and 
Head-Mounted  Displays  as  Alternatives  to  Multiple  Monitors  in  Information-Rich 
Environments.  In  environments  such  as  Navy  command  and  control,  supervision  and  multi¬ 
tasking  are  often  hindered  by  a  shortage  of  screen  space.  An  obvious  solution  is  to  use  more 
or  larger  monitors.  An  inexpensive  and  practical  alternative  is  to  use  an  interface  that  allows 
rapid  switching  between  workspaces  (screens  of  information),  providing,  in  effect,  a  large 
virtual  desktop.  One  potentially  effective  switching  interface  is  a  workspace  control  diagram: 
a  meaningful  arrangement  of  buttons,  each  of  which  provides  access  to  and  information  for  a 
particular  workspace.  In  the  first  experiment,  participants  best  performed  a  task  that  required 
frequent  navigation  among  workspaces  using  two  real  monitors  but  were  nearly  as  proficient 
when  using  one  monitor  with  a  workspace  control  diagram  operated  by  hot  keys.  In  the 
second  experiment,  participants  performed  concurrent  tracking  and  monitoring  tasks  about 
equally  using  four  monitors  with  all  workspaces  visible  and  using  two  monitors  with  a 
workspace  control  diagram.  Additionally,  the  authors  discuss  the  potential  of  a  much  more 
radical  workstation  design  alternative:  head-mounted  displays. 

Wickens,  C.D.,  Vincow,  M.A.,  Schopper,  A.W.,  and  Lincoln,  J.E.  (1997).  Computational 
Models  of  Human  Performance  in  the  Design  and  Layout  of  Controls  and  Displays.  This 
literature  describes  both  conceptual  and  empirically  derived  models  pertaining  to  the  layout 
and  grouping  of  information  within  display  panels  and  display  screens.  Several  of  the  models 
reviewed  are  also  applicable  to  the  spatial  organization  of  controls  and  displays.  Principles 
relating  to  centrality  of  location,  symmetry,  clutter,  frequency  of  use,  sequence  of  use,  relative 
importance,  and  relatedness  are  considered.  The  models  are  organized  into  three  areas:  (a) 
physical  arrangements  of  the  information  to  be  displayed  emphasizing  issues  such  as  overall 
and  local  screen  density,  grouping,  and  alignment,  (b)  interrelationships  derived  from  content 
and  task-related  factors  and  information  processing  considerations,  and  (c)  more 
comprehensive  conceptual  and  software -based  engineering  models  derived  from  groupings, 
density,  and  task-related  consideration;  relative  importance  of  various  components  and 
information  groups;  and  relative  distance  and  location. 
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4.  Conclusions  and  Recommendations 


A  survey  of  the  literature  related  to  operator  performance  using  multiple  displays  has  been 
completed  in  order  to  provide  feedback  on  the  domain,  type  and  quality  of  the  literature  and  to 
recommend  areas  of  future  research.  The  type  of  literature  identified  indicated  that  there  is 
more  relevant  literature  in  the  military  than  the  maritime  domain.  There  is  also  a  substantial 
base  of  literature  from  basic  research,  which  is  directed  mostly  at  visual  displays.  This 
literature  base,  however,  does  not  have  much  depth.  Much  of  the  literature  is  at  the 
experimental  phase  of  development.  This  reflects  the  lack  of  maturity  of  the  domain,  in  that 
less  new  ideas  have  been  generated  than  have  been  tested. 

In  cases  where  actual  evaluations  of  multiple  display  technologies  have  been  conducted  there 
are  a  number  of  concerns  with  the  quality  of  research,  particularly  as  a  result  of 
unrepresentative  subject  groups  and  tasks,  inappropriate  measures,  and  poor  interface  design 
of  displays. 

Human  factors  research  on  complex  processes,  both  their  control  and  their  fault  management 
and  diagnosis,  remains  an  immature  domain.  Many  of  the  recommendations  concerning 
displays  translate  directly  from  the  general  practice  of  good  human  factors,  and  this  approach 
has  been  implemented  with  some  degree  of  success.  Yet  it  is  equally  clear  that  great 
improvements  are  possible  by  combining  theories  of  cognitive  psychology  with  increasingly 
available  computer  and  display  technology.  This  area  of  cognitive  engineering  is  one  in 
which  relatively  little  engineering  psychology  research  has  been  conducted  to  assess  the 
utility  of  the  recommendations  made  by  such  researchers  as  Wickens,  Stanney  and  St.  John. 
The  potential  payoffs  of  this  research  appear  to  be  quite  high,  in  light  of  the  fact  that  systems 
are  evolving  toward  greater  complexity  with  increasing  levels  of  automation. 

In  order  to  extend  the  literature,  in  a  manner  that  would  generate  valid  guidance  for  future 
operator  performance  using  multiple  displays  in  the  maritime  environment,  it  is  recommended 
that  a  research  and  development  program  be  conducted,  as  follows: 

4.  Extend  the  literature  review  on  operator  performance  using  multiple  displays; 

5.  Conduct  maritime  multiple  display  experiments;  and 

6.  Coordinate  and  consolidate  multiple  display  research. 
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Atomic  Energy  of  Canada  Limited 
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Command,  Control  and  Communication 

Command,  Control,  Communications  and  Computer 
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Man-Systems  Integration  Standards 

NASA 

National  Aeronautics  and  Space  Administration 

NRC 

U.S.  Nuclear  Regulatory  Commission 
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National  Research  Council  Canada 
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Operator  Performance  and  Safety  Division 
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Public  Works  and  Government  Services  Canada 
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SPAWAR 

Space  and  Naval  Warfare  Systems  Command 

WM 

Working  Memory 
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